Image matching system using 3-dimensional object model, image matching method, and image matching program

ABSTRACT

Even when only a small number of reference images are available for each object, it is possible to search at high speed a reference image stored in a database from an input image of an object imaged with a different pose and a different illumination condition. A reference image matching result storage section ( 50 ) inputs reference images from a reference image storage section ( 70 ) and stores in advance results of matching of the input images with representative 3-dimensional object models of a representative 3-dimensional object model storage section ( 20 ). According to each representative 3-dimensional object model, image generation means ( 30 ) generates a comparison image having an input condition similar to the input image obtained from the image input means ( 10 ). Image matching means ( 40 ) calculates similarity between the input image and the image generated. Result matching means ( 60 ) calculates similarity between the matching result of the image matching means ( 40 ) and the reference image stored in the reference image matching result storage section ( 50 ), extracts reference images having similar matching results in the descending order of the similarity, and displays them on result display means ( 80 ).

TECHNICAL FIELD

The present invention relates to an image matching system using athree-dimensional object model, an image matching method, and an imagematching program. In particular, the present invention relates to animage matching system capable of retrieving a reference image stored ina database (DB) on the basis of an input image of an object (human face)picked up under a different pose and illumination condition, and itsimage matching method and image matching program.

BACKGROUND ART

An example of a conventional image matching system is described inTsutada etc., “Dictionary configuration method for discriminating personindependently of face direction,” The Transactions of the Institute ofElectronics, Information and Communication Engineers, D-II, Vol.J78-D-II, No. 11 (1995), pp. 1639-1649 (hereafter referred to as firstconventional technique). As shown in FIG. 26, an image matching systemaccording to the first image matching system includes an image inputsection 10, an image matching section 40, a result display section 80, areference image storage section 70, and a reference image registrationsection 75.

The conventional image matching system having such a configurationoperates as hereafter described.

Reference images of various objects (such as reference face images ofpersons) picked up are previously stored in the reference image storagesection 70 by the reference image registration section 75. However,reference images greatly change depending on the condition at the timeof imaging (conditions such as the pose and illumination). With respectto one object, therefore, a plurality of (a large number of) imagesimaged under various conditions are previously stored.

The image input section 10 is implemented by using, for example, acamera. The imaged input image is stored in a memory (not illustrated).The image matching section 40 compares the input image obtained from theimage input section 10 with each of reference images obtained from thereference image storage section 70, calculates similarities (or distancevalues) of respective features, and selects a reference image having thegreatest similarity (or the shortest distance) for each object. Eachimage is represented by gray level features. In the calculation ofsimilarity between features and the calculation of the distance value,for example, the normalized correlation and Euclidean distance are used.The result display section 80 displays a reference image of an objecthaving the greatest similarity selected from among the reference images,as a matching result (or displays candidate reference images in thedescending order of the similarity).

Another example of a conventional image matching system is described inJapanese Patent Application Laid-Open No. 2000-322577 (hereafterreferred to as second conventional technique). As shown in FIG. 28, theconventional image matching system includes an image input section 10,an image conversion section 35, a partial image matching section 45, aresult display section 80, a reference image registration section 75, arepresentative three-dimensional object model storage section 20, and athree-dimensional object model registration section 25.

The conventional image matching system having such a configurationoperates as hereafter described.

One or more representative three-dimensional object models obtained fromthe three-dimensional object model registration section 25 arepreviously stored in the representative three-dimensional object modelstorage section 20. As regards a partial region common to the inputimage obtained from the image input section 10 and each of the referenceimage obtained from the reference image storage section 70, the imageconversion section 35 converts the input image and/or the referenceimage so as to make the input condition (such as the pose condition) thesame by using a three-dimensional object model obtained from therepresentative three-dimensional object model storage section 20, andthereby generates partial images.

For example, as shown in FIG. 29, the partial region is a featureportion such as an eye, a nose or a mouth. By previously specifying afeature point with respect to each of the images and thethree-dimensional object models, correspondence can be taken. Thepartial image matching section 45 compares the converted input imageobtained from the image conversion section 35 with a partial image ofeach of the reference images, calculates respective averagesimilarities, and selects a reference image having the greatestsimilarity for each object. The result display section 80 displays anobject having the greatest similarity among the reference images, as amatching result.

DISCLOSURE OF INVENTION

The above-described conventional techniques have various problemsdescribed hereafter.

First, the above-described first and second conventional techniques havea problem that a large number of reference images of an object to beregistered, picked up under various conditions become necessary.

The reason is as follows: the input image is directly compared with areference image; if the input image pickup condition is not restricted,therefore, it is necessary to previously prepare reference images thatare close to the input image in pickup condition in order to cope with alarge number of pose and illumination conditions. As a matter of fact,however, there are infinite possibilities in the pose and illuminationconditions, and it is practically impossible to previously prepare alarge number of images associated with various conditions.

Secondly, in the second conventional technique, the input image or thereference image is converted so as to square them with each other inpose, and comparison is conducted. If the number of the reference imagesof a three-dimensional object model is not sufficient or the pose islargely different, therefore, distortion caused by the conversionbecomes large and matching cannot be conducted correctly, resulting in aproblem. There is also a problem that it is very difficult to square theillumination conditions with each other by conducting conversion and acommon region certainly needs to be present because images are comparedin the common region.

Thirdly, the conventional techniques have a problem that it takes aconsiderably long time to conduct matching.

The reason is as follows: in the conventional techniques, the inputimage is compared with a plurality of reference images of respectiveobjects; if the number of objects is M and the number of referenceimages of each object is L, therefore, image comparison must beconducted at least L□M times.

An object of the present invention is to provide an image matchingsystem, an image matching method, and an image matching program thatmake it possible to retrieve a reference image registered in a databaseon the basis of an input image as regards images picked under differentpose and illumination conditions every object, even when only a smallnumber of reference images are available.

Another object of the present invention is to provide an image matchingsystem, an image matching method, and an image matching program thatmake it possible to conduct matching with a small number of referenceimages of three-dimensional object models without conducting processingsuch as converting the input image or the reference image so as to makethe pose coincide and that makes it possible to conduct matching even ifa region common to the images is not present.

Another object of the present invention is to provide an image matchingsystem, an image matching method, and an image matching program thatmake it possible to conduct image matching without always generating acertain necessary number of three-dimensional objects with respect toall objects.

Still another object of the present invention is to provide an imagematching system, an image matching method, an image matching programthat make it possible to conduct retrieval at high speed even when adatabase has reference images concerning a large number of objectsregistered therein.

The present invention provides an image matching system for retrieving areference image similar to an input image, the image matching systemincluding means for making a match between the input image and aplurality of representative three-dimensional object models, means formaking a match between the reference image and the representativethree-dimensional object models, and means for retrieving the referenceimage similar to the input image by using a result of the match betweenthe input image and the representative three-dimensional object modelsand a result of the match between the reference image and therepresentative three-dimensional object models.

The image matching system may further include means for finding areference three-dimensional object model associated with the referenceimage similar to the input image, and means for newly retrieving thereference image similar to the input image by using the referencethree-dimensional object model and the input image.

The image matching system may further include means for finding areference three-dimensional object model associated with the referenceimage similar to the input image, conversion means for squaring an inputcondition of the input image with that of the reference image byconverting the input image and/or the reference image on the basis ofthe reference three-dimensional object model, and means for retrievingthe reference image associated with the input image by making a matchbetween the input image and reference image squared in input condition.

In the image matching system, the conversion means may previouslyconvert the reference image, and square an input condition of the inputimage with that of the reference image.

The image matching system may include image input means for inputtingthe input image, a representative three-dimensional object model storagesection for storing a plurality of representative three-dimensionalobject models, image generation means for generating at least onecomparison image close in input condition to the input image everyrepresentative three-dimensional object model on the basis of therepresentative three-dimensional object models stored in therepresentative three-dimensional object model storage section, imagematching means for calculating a similarity between the input image andeach of the comparison images generated by the image generation means,selecting a maximum similarity with respect to comparison imagesassociated with each representative three-dimensional object model, andregarding the maximum similarity as a similarity between the input imageand the representative three-dimensional object model, a reference imagestorage section for storing the reference images of objects, a referenceimage matching result storage section for storing similarities betweenthe reference images stored in the reference image storage section andrepresentative three-dimensional object models stored in therepresentative three-dimensional object model storage section, andresult matching means for extracting the reference image similar to theinput image on the basis of similarities between the input image and therepresentative three-dimensional object models calculated by the imagematching means and similarities between the reference images and therepresentative three-dimensional object models stored in the referenceimage matching result storage section.

The image matching system may further include three-dimensional objectmodel registration means for registering representativethree-dimensional object models in the representative three-dimensionalobject model storage section, reference image registration means forregistering reference images in the reference image storage section, andreference image matching result update means for conducting calculationof the similarities using the image matching means, on a combination ofa reference image and a representative three-dimensional object modelnewly generated by registration, when a new representativethree-dimensional object model is registered in the representativethree-dimensional object model storage section by the three-dimensionalobject model registration means, or when a new reference image isregistered in the reference image storage section by the reference imageregistration means, and adding a result of the calculation to results inthe reference image matching result storage section.

In the image matching system, the image matching means may calculate asimilarity between the input image and a representativethree-dimensional object model every partial region, the reference imagematching result storage section may store similarities between thereference images stored in the reference image storage section andrepresentative three-dimensional object models stored in therepresentative three-dimensional object model storage section, everypartial region, and the result matching means may extract the referenceimage similar to the input image on the basis of similarities betweenthe input image and the representative three-dimensional object modelscalculated by the image matching means every partial region andsimilarities between the reference images and the representativethree-dimensional object models, stored in the reference image matchingresult storage section every partial region.

In the image matching system, the result matching means may calculatesimilarities between similarities between the input image and therepresentative three-dimensional object models and similarities betweenthe reference images and the representative three-dimensional objectmodels, and in the calculation provide the resultant similarities withweights on the basis of candidate precedence of similarities between theinput image and the comparison images and the representativethree-dimensional object models.

The image matching system may include image input means for inputtingthe input image, a representative three-dimensional object model storagesection for storing a plurality of representative three-dimensionalobject models, image generation means for generating at least onecomparison image close in input condition to the input image everyrepresentative three-dimensional object model on the basis of therepresentative three-dimensional object models stored in therepresentative three-dimensional object model storage section, imagematching means for calculating a similarity between the input image andeach of the comparison images generated by the image generation means,selecting a maximum similarity with respect to comparison imagesassociated with each representative three-dimensional object model, andregarding the maximum similarity as a similarity between the input imageand the representative three-dimensional object model, a reference imagestorage section for storing the reference images of objects, a referenceimage matching result storage section for storing similarities betweenthe reference images stored in the reference image storage section andrepresentative three-dimensional object models stored in therepresentative three-dimensional object model storage section, resultmatching means for extracting the reference image similar to the inputimage on the basis of similarities between the input image and therepresentative three-dimensional object models calculated by the imagematching means and similarities between the reference images and therepresentative three-dimensional object models stored in the referenceimage matching result storage section, a reference three-dimensionalobject model storage section for storing reference three-dimensionalobject models associated with the reference images stored in thereference image storage section, second image generation means forobtaining reference three-dimensional object models associated withreference images extracted from the result matching means, from thereference three-dimensional object model storage section, and generatingat least one second comparison image close in input condition to theinput image every reference three-dimensional object model on the basisof the obtained reference three-dimensional object models, and secondimage matching means for calculating similarities between the inputimage and second comparison images generated by the second imagegeneration means, selecting a maximum similarity from among secondcomparison images associated with each of the referencethree-dimensional object models, and regarding the maximum similarity asa similarity between the input image and the reference three-dimensionalobject model.

The image matching may further include three-dimensional object modelregistration means for registering representative three-dimensionalobject models in the representative three-dimensional object modelstorage section, reference image registration means for registeringreference images in the reference image storage section, and referenceimage matching result update means for conducting calculation of thesimilarities using the image matching means, on a combination of areference image and a representative three-dimensional object modelnewly generated by registration, when a new representativethree-dimensional object model is registered in the representativethree-dimensional object model storage section by the three-dimensionalobject model registration means, or when a new reference image isregistered in the reference image storage section by the reference imageregistration means, and adding a result of the calculation to results inthe reference image matching result storage section, andthree-dimensional object model generation means responsive toregistration of a similarity between the reference image and therepresentative three-dimensional object model in the reference imagematching result storage section conducted by the reference imagematching result update means, for generating the referencethree-dimensional object model associated with the reference image bycombining the representative three-dimensional object models stored inthe representative three-dimensional object model storage section on thebasis of the similarity, and registering the generated referencethree-dimensional object model in the reference three-dimensional objectmodel storage section.

In the image matching system, the three-dimensional object modelgeneration means may generate a reference three-dimensional object modelassociated with each reference image by combining representativethree-dimensional object models stored in the representativethree-dimensional object model storage section every partial region, onthe basis of similarities obtained every partial region betweenreference images stored in the reference image storage section andrepresentative three-dimensional object models stored in therepresentative three-dimensional object model storage section, andregister the generated reference three-dimensional object model in thereference three-dimensional object model storage section.

In the image matching system, the image matching means may calculate asimilarity between the input image and a representativethree-dimensional object model every partial region, the reference imagematching result storage section may store similarities between thereference images stored in the reference image storage section andrepresentative three-dimensional object models stored in therepresentative three-dimensional object model storage section, everypartial region, and the result matching means may extract the referenceimage similar to the input image on the basis of similarities betweenthe input image and the representative three-dimensional object modelscalculated by the image matching means every partial region andsimilarities between the reference images and the representativethree-dimensional object models, stored in the reference image matchingresult storage section every partial region.

In the image matching system, the result matching means may calculatesimilarities between similarities between the input image and therepresentative three-dimensional object models and similarities betweenthe reference images and the representative three-dimensional objectmodels, and in the calculation, provide the resultant similarities withweights on the basis of candidate precedence of similarities between theinput image and the comparison images and the representativethree-dimensional object models.

The image matching system may include image input means for inputtingthe input image, a representative three-dimensional object model storagesection for storing a plurality of representative three-dimensionalobject models, image generation means for generating at least onecomparison image close in input condition to the input image everyrepresentative three-dimensional object model on the basis of therepresentative three-dimensional object models stored in therepresentative three-dimensional object model storage section, imagematching means for calculating a similarity between the input image andeach of the comparison images generated by the image generation means,selecting a maximum similarity with respect to comparison imagesassociated with each representative three-dimensional object model, andregarding the maximum similarity as a similarity between the input imageand the representative three-dimensional object model, a reference imagestorage section for storing the reference images of objects, a referenceimage matching result storage section for storing similarities betweenthe reference images stored in the reference image storage section andrepresentative three-dimensional object models stored in therepresentative three-dimensional object model storage section, resultmatching means for extracting the reference image similar to the inputimage on the basis of similarities between the input image and therepresentative three-dimensional object models calculated by the imagematching means and similarities between the reference images and therepresentative three-dimensional object models stored in the referenceimage matching result storage section, a reference three-dimensionalobject model storage section for storing reference three-dimensionalobject models associated with the reference images stored in thereference image storage section, image conversion means for obtainingreference three-dimensional object models associated with referenceimages extracted from the result matching means, from the referencethree-dimensional object model storage section, squaring an inputcondition of the input image with that of the reference image extractedby the result matching means by converting the input image and/or thereference image extracted by the result matching means, on the basis ofthe obtained reference three-dimensional object models, and generatingpartial images respectively of the input image and the reference imagesquared in input condition with each other, and partial image matchingmeans for calculating a similarity between the partial image of theinput image and the partial image of the reference image generated bythe image conversion means.

The image matching system may further include three-dimensional objectmodel registration means for registering representativethree-dimensional object models in the representative three-dimensionalobject model storage section, reference image registration means forregistering reference images in the reference image storage section, andreference image matching result update means for conducting calculationof the similarities using the image matching means, on a combination ofa reference image and a representative three-dimensional object modelnewly generated by registration, when a new representativethree-dimensional object model is registered in the representativethree-dimensional object model storage section by the three-dimensionalobject model registration means, or when a reference image is registeredin the reference image storage section by the reference imageregistration means, and adding a result of the calculation to results inthe reference image matching result storage section, andthree-dimensional object model generation means responsive toregistration of a similarity between the reference image and therepresentative three-dimensional object model in the reference imagematching result storage section conducted by the reference imagematching result update means, for generating the referencethree-dimensional object model associated with the reference image bycombining the representative three-dimensional object models stored inthe representative three-dimensional object model storage section on thebasis of the similarity, and registering the generated referencethree-dimensional object model in the reference three-dimensional objectmodel storage section.

In the image matching system, the three-dimensional object modelgeneration means may generate a reference three-dimensional object modelassociated with each reference image by combining representativethree-dimensional object models stored in the representativethree-dimensional object model storage section every partial region, onthe basis of similarities obtained every partial region betweenreference images stored in the reference image storage section andrepresentative three-dimensional object models stored in therepresentative three-dimensional object model storage section, andregister the generated reference three-dimensional object model in thereference three-dimensional object model storage section.

In the image matching system, the image matching means may calculate asimilarity between the input image and a representativethree-dimensional object model every partial region, the reference imagematching result storage section may store similarities between thereference images stored in the reference image storage section andrepresentative three-dimensional object models stored in therepresentative three-dimensional object model storage section, everypartial region, and the result matching means may extract the referenceimage similar to the input image on the basis of similarities betweenthe input image and the representative three-dimensional object modelscalculated by the image matching means every partial region andsimilarities between the reference images and the representativethree-dimensional object models, stored in the reference image matchingresult storage section every partial region.

In the image matching system, the result matching means may calculatesimilarities between similarities between the input image and therepresentative three-dimensional object models and similarities betweenthe reference images and the representative three-dimensional objectmodels, and in the calculation, provide the resultant similarities withweights on the basis of candidate precedence of similarities between theinput image and the comparison images and the representativethree-dimensional object models.

In the image matching system, the object may be a human face.

According to the present invention, effects described hereafter areachieved.

A first effect will now be described. With respect to an input image ofan object picked up under a different input condition such as adifferent pose and illumination condition a reference image of the sameobject can be retrieved, even if only one reference image or a smallnumber of reference images are present. Furthermore, matching can beconducted with a small number of reference images of three-dimensionalobject models, without conducting processing such as converting theinput image or the reference image so as to make their poses coincidewith each other. In addition, matching can be conducted even if a regioncommon to the images is not present. Furthermore, image matching becomespossible without always generating a predetermined number ofthree-dimensional object models for every object.

The reason is that a reference image is retrieved by comparing a resultof matching between the input image and representative three-dimensionalobject models with a result of matching between reference images andrepresentative three-dimensional object models. The reason is also thata reference three-dimensional object model is generated by combiningrepresentative three-dimensional object models and subjected tomatching.

A second effect is that a reference image can be retrieved at high speedwith respect to an input image.

The reason is that matching of representative three-dimensional objectmodels less than the objects is conducted and image matching isconducted by using the calculation of similarity of the matching result.Even when conducting matching with a reference three-dimensional objectmodel, reference images having high similarity are extracted by usingrepresentative three-dimensional object models, and then matching ofhigh precedence candidates with reference three-dimensional objectmodels is conducted.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram showing a configuration of an image matchingsystem according to a first embodiment of the present invention;

FIG. 2 is a flow char showing operation conducted at the time ofmatching in a first embodiment;

FIG. 3 is a diagram showing a concrete example of a representativethree-dimensional object model in a first embodiment;

FIG. 4 is a diagram showing concrete examples of a reference image in afirst embodiment;

FIG. 5 is a diagram showing a concrete example of a reference imagematching result in a first embodiment;

FIG. 6 is a diagram showing a concrete example of an input image in afirst embodiment;

FIG. 7 is a diagram showing a concrete example of an input imagematching result in a first embodiment;

FIG. 8 is a diagram showing a concrete example of result matching in afirst embodiment;

FIG. 9 is a block diagram showing a configuration of an image matchingsystem according to a second embodiment of the present invention;

FIG. 10 is a flow chart showing operation conducted at the time ofthree-dimensional object model registration in a second embodiment;

FIG. 11 is a flow chart showing operation conducted at the time ofreference image registration in a second embodiment;

FIG. 12 is a diagram showing a concrete example of a matching result ofa three-dimensional object model registered in a second embodiment;

FIG. 13 is a diagram showing a concrete example of update of a referenceimage matching result in a second embodiment;

FIG. 14 is a diagram showing a concrete example of a matching result ofa registered reference image in a second embodiment;

FIG. 15 is a diagram showing a concrete example of update of a referenceimage matching result in a second embodiment;

FIG. 16 is a block diagram showing a configuration of an image matchingsystem according to a third embodiment of the present invention;

FIG. 17 is a flow chart showing operation conducted at the time ofmatching in a third embodiment;

FIG. 18 is a diagram showing concrete examples of a referencethree-dimensional object model in a third embodiment;

FIG. 19 is a diagram showing a concrete example of a reference imagematching result in a third embodiment;

FIG. 20 is a block diagram showing a configuration of an image matchingsystem according to a fourth embodiment of the present invention;

FIG. 21 is a flow chart showing operation conducted at the time ofthree-dimensional object model registration in a fourth embodiment;

FIG. 22 is a flow chart showing operation conducted at the time ofreference image registration in a fourth embodiment;

FIG. 23 is a block diagram showing a configuration of an image matchingsystem according to a fifth embodiment of the present invention;

FIG. 24 is a flow chart showing operation conducted at the time ofmatching in a fifth embodiment;

FIG. 25 is a block diagram showing a configuration of an image matchingsystem according to a sixth embodiment of the present invention;

FIG. 26 is a block diagram showing a configuration of an image matchingsystem according to a first conventional technique;

FIG. 27 is a diagram showing a concrete example of coordinates of athree-dimensional object model;

FIG. 28 is a block diagram showing a configuration of an image matchingsystem according to a second conventional technique; and

FIG. 29 is a diagram showing a concrete example of a partial region of asecond conventional technique.

BEST MODE FOR CARRYING OUT THE INVENTION

Hereafter, embodiments of the present invention will be described indetail with reference to the drawings.

With reference to FIG. 1, an image matching system according to a firstembodiment of the present invention includes an image input section 10,an image generation section 30, an image matching section 40, a resultmatching section 60, a result display section 80, a reference imagestorage section 70, a representative three-dimensional object modelstorage section 20, and a reference image matching result storagesection 50.

Representative three-dimensional object models (three-dimensional shapesof objects and textures on object surfaces) are previously registered inthe representative three-dimensional object model storage section 20.The three-dimensional object models can be generated by using, forexample, a three-dimensional shape measurement apparatus described inJapanese Patent Application Laid-Open No. 2001-12925 or an apparatusthat reconstructs a three-dimensional shape from a plurality of imagespicked up with a large number of cameras and that is described inJapanese Patent Application Laid-Open No. 9-91436.

As shown in FIG. 27, the three-dimensional object model has a shapeP_(Q)(x, y, z) and a texture T_(Q)(R, G, B) in a three-dimensional space(x, y, z) at an object surface, as information. Q represents an index ofa point on the object surface, and corresponds to coordinates of a pointQ(s,t) obtained by projecting a point on the object surface onto asphere centering around the center of gravity of the object from thecenter of gravity. For the purpose of matching, CG images for learningunder various illumination conditions are previously generated by usingthree-dimensional object models and computer graphics, and a basic imagegroup is derived by conducting principal component analysis on the CGimages for learning.

The image generation section 30 generates a plurality of comparisonimages close in illumination condition to an input image obtained fromthe image input section 10, while supposing a pose condition on thebasis of a representative three-dimensional object model obtained fromthe representative three-dimensional object model storage section 20.Here, the generation of the comparison image close in illuminationcondition to the input image can be implemented by conducting coordinateconversion on a basic image group obtained beforehand on the basis ofthe supposed pose condition and obtaining coefficients of a linear sumby using the least square method so as to make the linear sum of thebasic images subjected to the coordinate conversion close to the inputimage.

A technique for generating a comparison image close to the input imagefrom the three-dimensional object model is described in, for example,“Face matching using automatic correction of both illumination conditionand pose,” Technical Report of the Institute of Electronics, Informationand Communication Engineers, Vol. 101, No. 524, PRMU2001-153˜175 (2001),pp. 59-64.

The image matching section 40 estimates the pose by comparing the inputimage obtained from the image input section 10 with each of comparisonimages obtained from the image generation section 30, calculating asimilarity between the input image and each of the comparison images,and selecting a comparison image having the greatest similarity for eachobject.

The image generation section 30 and the image matching section 40handles each of the reference images stored in the reference imagestorage section 70 serving as a database (DB) that stores referenceimages as the input image and matches representative three-dimensionalobject models stored in the representative three-dimensional objectmodel storage section 20 with each of the reference images. Resultsobtained by the matching are previously stored in the reference imagematching result storage section 50.

The result matching section 60 compares a result of matching of theinput image obtained from the image input section 10 conducted by theimage generation section 30 and the image matching section 40 with thematching result of each reference image in the reference image matchingresult storage section 50, and extracts reference images having similarmatching results, in the descending order of the similarity. The resultdisplay section 80 displays an object having the greatest similarity asa matching result.

Reference images which are two-dimensional images of an object to beretrieved are registered in the reference image storage section 70. Asfor the reference images, there are no restrictions to the inputcondition including the illumination and pose. At least one image isregistered every object (retrieval subject).

A plurality of representative three-dimensional object models are storedin the representative three-dimensional object model storage section 20.

General operation of the first embodiment will now be described indetail with reference to FIG. 1 and a flow chart shown in FIG. 2.

At the time of input image matching, an input image is first obtained bythe input image section 10 (step 100 in FIG. 2). Subsequently, the imagegeneration section 30 generates comparison images that are close in theinput condition such as the pose and illumination to the input image,i.e., comparison images that facilitate comparison, with respect to eachof the representative three-dimensional object models stored in therepresentative three-dimensional object model storage section 20 (step101).

The image matching section 40 finds similarity between the input imageand each of the comparison images (step 102). The result matchingsection 60 calculates a similarity between the matching result and amatching result of each of the reference image stored in the referenceimage matching result storage section 50, and extracts reference imageshaving similar matching results, in the descending order of thesimilarity (step 103). Finally, a reference image having high similarityis displayed (step 104).

Effects of the first embodiment which has the configuration and whichoperates as described above will now be described.

The first embodiment has the configuration in which reference images areretrieved by comparing a result of matching between the input image andthe representative three-dimensional object models with a result ofmatching between reference images and the representativethree-dimensional object models. Even when only one reference image or asmall number of reference images are present every object, therefore,reference images can be retrieved with respect to an input image of anobject picked up under a different condition concerning the pose andillumination.

The present embodiment has a configuration in which image matching isconducted by conducting matching with representative three-dimensionalobject models which are less than objects and conducting similaritycalculation on the results of the matching. This makes fast retrievalpossible. Since the time taken for the similarity calculation ofmatching results is shorter than the time taken for the image matching,the retrieval time depends on the number of image matching operations.For example, if the number N of the representative three-dimensionalobject models is N=M/100 where M is the number of objects (the number ofreference images), then the number of required image matching operationsis L×N=L×M/100 where L is the number of comparison images for eachrepresentative three-dimensional object model generated in the imagegeneration section 30. Thus, retrieval can be conducted with the numberof times of matching as small as 1/100 of that in the conventionaltechnique.

Operation of the first embodiment will now be described with referenceto FIGS. 3 to 8 which show concrete embodiments.

As shown in FIG. 3, N representative three-dimensional object modelsC_(j) (j=1, 2, . . . , N) are stored in the representativethree-dimensional object model storage section 20. As shown in FIG. 4, Mreference images R_(i) (i=1, 2, . . . , M) respectively of the objectsare stored in the reference image storage section 70 (A plurality ofreference images may be present for each of the objects. In the ensuingdescription, however, it is supposed that one reference image is storedevery object).

As shown in FIG. 5, a result (similarity S_(ij)) of matching of each ofthe reference images R_(i) with the representative three-dimensionalobject model C_(j) is stored in the reference image matching resultstorage section 50 by processing at the time of reference imageregistration. (In FIG. 5, the matching results are shown in thedescending order of the similarity. As a matter of fact, however, thematching results may be stored in the order of model.)

It is supposed that an input image I(u, v) as shown in FIG. 6 isobtained by the image input section 10 at the time of matching of theinput image (step 100 in FIG. 2). Subsequently, the image generationsection 30 generates L comparison images G_(jk)(u, v) (j=, N, k=1, . . ., L) which are close in input condition such as the pose andillumination to the input image, with respect to each of therepresentative three-dimensional object models C_(j) (j=1, . . . , N) inthe representative three-dimensional object model storage section 20(step 101).

In addition, the image matching section 40 finds a similarity S(I,G_(jk)) between the input image I(u, v) and each of the comparisonimages G_(jk)(u, v), and finds a maximum similarity S_(0j) _(—)=max_(k)S(I, G_(jk)) every representative three-dimensional object model(step 102). The matching results (similarities) S_(0j) become, forexample, as shown in FIG. 7.

The result matching section 60 calculates a similarity D_(i)=D(S_(0j),S_(ij)) between the matching result S_(0j) and the matching resultS_(ij) of each of the reference images in the reference image matchingresult storage section 50, and extracts reference images in thedescending order of the similarity D_(i) of the matching result (step103). The result of the extraction becomes, for example, as shown inFIG. 8. As reference images having a high possibility of being an imageof the same object as the input image, R₁, R₅ and R₂ are obtained in thecited order. Finally, the reference images having high similarities aredisplayed (step 104).

As the calculation method of the similarity D_(i)(S_(0j), S_(ij)) of thematching result, normalized correlation, rank_correlation, or the likecan be used. The rank correlation is correlation of candidate precedenceof the matching result. Denoting the candidate precedence of thematching result S_(0j) of the input image by A_(0j), it follows thatA_(0.2)=1, A_(0.6)=2 and A_(0.3)=3 in the case of the matching resultshown in FIG. 7. Denoting candidate precedence of the matching resultS_(ij) of each of the reference images by A_(ij), for example, theSpearman's rank correlation can be obtained according to the expression1−6Σ_(j)(A_(0j)−A_(ij))² /{N(N ²−1)}.

In the similarity calculation, the similarities may be calculated afterconducting variable conversion on the variables (such as S_(0j), S_(ij)and A_(0j), A_(ij)). The similarities may be calculated by weightingvariables with weights g(A_(0j), A_(ij)) based on the candidateprecedence A_(0j) and/or A_(ij). For example, specific gravities of highprecedence candidates become great by setting g(A_(0j),A_(ij))=1/(A_(0j)+A_(ij)) and replacing the similarities S_(0j) andS_(ij) respectively by S_(0j)/(A_(0j)+A_(ij)) andS_(ij)/(A_(0j)+A_(ij)). The similarities may be calculated with lowprecedence candidates excluded.

An image matching system according to a second embodiment of the presentinvention will now be described with reference to FIGS. 9 to 15. Animage matching system according to the second embodiment of the presentinvention includes an image input section 10, an image generationsection 30, an image matching section 40, a result matching section 60,a result display section 80, a reference image storage section 70, areference image registration section 75, a representativethree-dimensional object model storage section 20, a three-dimensionalobject model registration section 25, a reference image matching resultstorage section 50, and a reference image matching result update section55. This configuration is obtained by adding the reference imageregistration section 75, the three-dimensional object model registrationsection 25, and the reference image matching result update section 55 tothe configuration of the first embodiment.

As to components in the image matching system according to the secondembodiment that are the same as those in the first embodiment,description will be omitted. The components added in the presentembodiment will be described.

The three-dimensional object model registration section 25 registers anew representative three-dimensional object model (a three-dimensionalshape of an object and a texture on the object surface) in therepresentative three-dimensional object model storage section 20.

When at the time of registration a representative three-dimensionalobject model is registered in the representative three-dimensionalobject model storage section 20 by the three-dimensional object model25, and when a reference image is registered in the reference imagestorage section 70 by the reference image registration section 75, thereference image matching result update section 55 conducts matchingoperation on a combination of a new reference image and a representativethree-dimensional object model by using the image generation section 30and the image matching section 40, and adds a result of the matching todata in the reference image matching result storage section 50.

The reference image registration section 75 registers reference imageswhich are two-dimensional images of an object to be retrieved, in thereference image storage section 70. As for the reference images, thereare no restrictions to the input condition including the illuminationand pose. At least one image is registered every object (retrievalsubject).

By the way, the three-dimensional object model registration section 25is the same as the three-dimensional object model registration section25 in the second conventional technique shown in FIG. 28. Representativethree-dimensional object models obtained from the three-dimensionalobject model registration section 25 are previously stored in therepresentative three-dimensional object model storage section 20.

General operation of the second embodiment will now be described indetail with reference to FIG. 9 and flow charts shown in FIGS. 2, 10 and11.

Operation conducted at the time of matching of the input image iscompletely the same as the operation of the first embodiment shown inFIG. 2.

At the time of input image matching, an input image is first obtained bythe input image section 10 (step 100 in FIG. 2). Subsequently, the imagegeneration section 30 generates comparison images that are close in theinput condition such as the pose and illumination to the input image,i.e., comparison images that facilitate comparison, with respect to eachof the representative three-dimensional object models stored in therepresentative three-dimensional object model storage section 20 (step101).

The image matching section 40 finds a similarity between the input imageand each of the comparison images (step 102). The result matchingsection 60 calculates a similarity between the matching result and thematching result of each of the reference images stored in the referenceimage matching result storage section 50, and extracts reference imageshaving similar matching results, in the descending order of thesimilarity (step 103). Finally, a reference image having high similarityis displayed (step 104).

Operation conducted at the time of representative three-dimensionalobject model registration and operation conducted at the time ofreference image registration will now be described.

At the time of representative three-dimensional object model (athree-dimensional shape of an object and a texture on the objectsurface) registration, the three-dimensional object model registrationsection 25 first registers a new representative three-dimensional objectmodel in the representative three-dimensional object model storagesection 20 (step 200 in FIG. 10).

Subsequently, the reference image matching result update section 55sends each of the reference images stored in the reference image storagesection 70 to the image input section 10 as the input image. Thereference image matching result update section 55 conducts matching, inthe image matching section 40, of each of the reference images withcomparison images generated by the image generation section 30 on thebasis of the registered representative three-dimensional object model,and finds a similarity (step 201). Finally, the reference image matchingresult update section 55 adds a result of the matching to each of thematching results of the reference images stored in the reference imagematching result storage section 50 (step 202).

At the time of reference image registration, the reference imageregistration section 75 first registers a new reference image in thereference image storage section 70 (step 210 in FIG. 11).

Subsequently, the reference image matching result update section 55sends the reference image registered in the reference image storagesection 70 to the image input section 10 as the input image. Thereference image matching result update section 55 conducts matching, inthe image matching section 40, of the reference image with comparisonimages generated by the image generation section 30 on the basis of therepresentative three-dimensional object model stored in therepresentative three-dimensional object model storage section 20, andfinds similarities (step 211). Finally, the reference image matchingresult update section 55 adds a result of the matching to the referenceimage matching result storage section 50 (step 212).

Effects of the second embodiment which has the configuration and whichoperates as described above will now be described.

The second embodiment has the configuration in which reference imagesare retrieved by comparing a result of matching between the input imageand the representative three-dimensional object models with a result ofmatching between reference images and the representativethree-dimensional object models. Even when only one reference image or asmall number of reference images are present every object, therefore,reference images can be retrieved with respect to an input image of anobject picked up under a different condition concerning the pose andillumination.

The present embodiment has a configuration in which image matching isconducted by conducting matching with representative three-dimensionalobject models which are less than objects and conducting similaritycalculation on the results of the matching. This makes fast retrievalpossible. Since the time taken for the similarity calculation ofmatching results is shorter than the time taken for the image matching,the retrieval time depends on the number of image matching operations.For example, if the number N of the representative three-dimensionalobject models is N=M/100 where M is the number of objects (the number ofreference images), then the number of required image matching operationsis L×N=L×M/100 where L is the number of comparison images for eachrepresentative three-dimensional object model generated in the imagegeneration section 30. Thus, retrieval can be conducted with the numberof times of matching as small as 1/100 of that in the conventionaltechnique.

Operation of the second embodiment will now be described with referenceto concrete embodiments.

As shown in FIG. 3, N representative three-dimensional object modelsC_(j) (j=1, 2, . . . , N) are stored in the representativethree-dimensional object model storage section 20 in the same way as thefirst embodiment. As shown in FIG. 4, M reference images R_(i) (i=1, 2,. . . , M) of the objects are stored in the reference image storagesection 70. As shown in FIG. 5, a result (similarity S_(ij)) of matchingof each reference image R_(i) with the representative three-dimensionalobject model C_(j) is stored in the reference image matching resultstorage section 50 by processing at the time of reference imageregistration.

It is supposed that an input image I(u, v) as shown in FIG. 6 isobtained by the image input section 10 at the time of matching of theinput image (step 100 in FIG. 2). Subsequently, the image generationsection 30 generates L comparison images G_(jk)(u, v) (j=1, . . . N,k=1, . . . , L) which are close in input condition such as the pose andillumination to the input image, with respect to each of therepresentative three-dimensional object models C_(j) (j=1, . . . , N) inthe representative three-dimensional object model storage section 20(step 101).

In addition, the image matching section 40 finds a similarity S(I,G_(jk)) between the input image I(u, v) and each of the comparisonimages G_(jk)(u, v), and finds a maximum similarity S_(oj)=max_(k)S(I,G_(jk)) every representative three-dimensional object model (step 102).The matching results (similarities) S_(oj) become, for example, as shownin FIG. 7.

The result matching section 60 calculates similarities D_(i)=D(S_(0j),S_(ij)) between the matching result S_(0j) and the matching resultsS_(ij) of the reference images in the reference image matching resultstorage section 50, and extracts reference images in the descendingorder of the similarity D_(i) of the matching result (step 103). Theresult of the extraction becomes, for example, as shown in FIG. 8. Asreference images having a high possibility of being an image of the sameobject as the input image, R₁, R₅ and R₂ are obtained in the citedorder. Finally, the reference images having high similarities aredisplayed (step 104).

At the time of representative three-dimensional object modelregistration, the three-dimensional object model registration section 25first registers a new representative three-dimensional object model. IfN=50 representative three-dimensional object models are alreadyregistered in the representative three-dimensional object model storagesection 20, the three-dimensional object model registration section 25registers a new fifty-first representative three-dimensional objectmodel C₅₁ (step 200 in FIG. 10).

Subsequently, the reference image matching result update section 55sends each reference image R_(i) stored in the reference image storagesection 70 to the image input section 10 as the input image. Thereference image matching result update section 55 conducts matching ofthe each reference image R_(i) with the registered representativethree-dimensional object model C₅₁ by using the image generation section30 and the image matching section 40, and finds each similarityS_(i,51)=max_(k)S(R_(i), G_(51,k)) (step 201).

The matching result (similarity) S_(i,51) becomes, for example, as shownin FIG. 12. Finally, as shown in FIG. 13, the reference image matchingresult update section 55 adds the matching result to the matchingresults of each of the reference images stored in the reference imagematching result storage section 50 (step 202).

At the time of reference image registration, the reference imageregistration section 75 first registers a new reference image in thereference image storage section 70. If M=100 reference images arealready registered in the reference image storage section 70, thereference image registration section 75 registers a new hundred firstreference image R₁₀₁ in the reference image storage section 70 (step 210in FIG. 11).

Subsequently, the reference image matching result update section 55sends the reference image R₁₀₁ registered in the reference image storagesection 70 to the image input section 10 as the input image. Thereference image matching result update section 55 conducts matching ofthe reference image R₁₀₁ with each three-dimensional object model C_(j)stored in the representative three-dimensional object model storagesection 20 by using the image generation section 30 and the imagematching section 40, and finds each similarity S_(101,j)=max_(k)S(R₁₀₁,G_(jk)) (step 211).

The matching result (similarity) becomes, for example, as shown in FIG.14. Finally, the reference image matching result update section 55 addsthe matching result to the reference image matching result storagesection 50 (step 212).

An image matching system according to a third embodiment of the presentinvention will now be described in detail with reference to FIGS. 16 to19.

With reference to FIG. 16, an image matching system according to thethird embodiment of the present invention includes an image inputsection 10, an image generation section 30, an image matching section40, a result matching section 60, a second image generation section 31,a second image matching section 41, a result display section 80, areference image storage section 70, a representative three-dimensionalobject model storage section 20, a reference image matching resultstorage section 50, and a reference three-dimensional object modelstorage section 21.

These components nearly operate as described below. The image inputsection 10, the image generation section 30, the image matching section40, the result matching section 60, the result display section 80, thereference image storage section 70, and the representativethree-dimensional object model storage section 20 conduct the sameprocessing as the processing conducted in the first embodiment of thepresent invention shown in FIG. 1.

Reference three-dimensional object models associated with the referenceimage are previously stored in the reference three-dimensional objectmodel storage section 21. The reference three-dimensional object modelscan be generated by combining representative three-dimensional objectmodels stored in the representative three-dimensional object modelstorage section 20 on the basis of information concerning the referenceimage matching result registered in the reference image matching resultstorage section 50. Or if three-dimensional object models of the sameobject as the reference image are previously generated by athree-dimensional shape measurement apparatus in the same way as theabove-described representative three-dimensional object modelregistration, the three-dimensional object models may be used.

The second image generation section 31 generates comparison images thatare close in input condition such as the pose and illumination conditionto the input image obtained from the image input section 10, for areference image that is a high precedence candidate in the matchingresults obtained from the result matching section 60, on the basis ofeach of the reference three-dimensional object model associated with thereference image obtained from the reference three-dimensional objectmodel storage section 21.

The second image matching section 41 compares the input image obtainedfrom the image input section 10 with each of the comparison imagesobtained from the second image generation section 31, and calculateseach similarity.

General operation of the third embodiment will now be described indetail with reference to FIG. 16 and a flow chart shown in FIG. 17.

At the time of input image matching, steps 100, 101, 102 and 103 shownin FIG. 17 are the same as the operation conducted in the firstembodiment shown in FIG. 2.

The second image generation section 31 generates comparison images thatare close in input condition such as the pose and illumination conditionto the input image obtained from the image input section, for areference image that is a high precedence candidate in the matchingresults obtained from the result matching section 60, on the basis ofeach of the reference three-dimensional object model associated with thereference image obtained from the reference three-dimensional objectmodel storage section 21 (step 111).

The second image matching section 41 compares the input image obtainedfrom the image input section with each of the comparison images obtainedfrom the second image generation section 31, and calculates eachsimilarity (step 112). Finally, the reference image having highsimilarity is displayed (step 104).

Effects of the third embodiment which has the configuration and whichoperates as described above will now be described.

The third embodiment has the configuration in which the referencethree-dimensional object model generated by combining representativethree-dimensional object models is matched. Even when only one referenceimage is present every object, therefore, reference images can beretrieved by using the reference three-dimensional object models, withrespect to an input image of an object picked up under a differentcondition concerning the pose and illumination.

The present embodiment has a configuration in which reference imageshaving high similarity are extracted by using a representativethree-dimensional object model and then matching of the referencethree-dimensional object model with high precedence candidates isconducted. As a result, reference images can be retrieved at high speed.

Operation of the third embodiment will now be described with referenceto concrete embodiments.

In the same way as the operation of the first embodiment, representativethree-dimensional object models C_(j) (j=1, 2, . . . , N) as shown inFIG. 3 are stored in the representative three-dimensional object modelstorage section 20. Reference images R_(i) (i=1, 2, . . . , M)respectively of the objects as shown in FIG. 4 are stored in thereference image storage section 70. A result (similarity) S_(ij) ofmatching of each of the reference images R_(i) with the representativethree-dimensional object model C_(j) as shown in FIG. 5 is stored in thereference image matching result storage section 50.

In addition, M reference three-dimensional object models Bi (i=1, 2, . .. , M) associated with the reference image R_(i) are previously storedin the reference three-dimensional object model storage section 21 asshown in FIG. 18.

It is supposed that an input image I(u, v) as shown in FIG. 6 isobtained by the image input section 10 at the time of matching of theinput image (step 100 in FIG. 16). According to the same processing asthe operation in the first embodiment, R₁, R₅ and R₂ are obtained in thecited order as reference images having a high possibility of being animage of the same object as the input image as shown in FIG. 8 by theimage generation section 30, the image matching section 40, the resultmatching section 60, and the result matching section 60 (steps 101, 102and 103).

With respect to, for example, the reference images R₁, R₅ and R₂ whichare three high precedence candidates in the matching result obtainedfrom the result matching section 60, the second image generation section31 acquires associated reference three-dimensional object models B₁, B₅and B₂ from the reference three-dimensional object model storage section21, and generates comparison images H_(jk)(u, v) (j=1, 5, 2, k=1, . . ., L) which are close in input condition such as the pose andillumination to the input image obtained from the image input section 10(step 111). The generation of the comparison images H_(jk)(u, v) isconducted by using a method similar to the step S101. In other words,the second image generation section 31 generates L comparison imagesH_(jk)(u, v) (j=1, 5, 2, k=1, . . . , L) which are close in inputcondition such as the pose and illumination to the input image, withrespect to the reference three-dimensional object models B_(j) (j=1, 5,2) in the reference three-dimensional object model storage section 21.The second image matching section 41 finds a similarity S(I, H_(jk))between the input image I(u, v) and each comparison image H_(jk)(u, v),and finds a maximum similarity S_(oj)=max_(k)S(I, H_(jk)) every model(step 112).

The matching results become, for example, as shown in FIG. 19. IfS₀₅>S₀₁>S₀₂, then R₅, R₁ and R₂ are obtained in the cited order asreference images having a high possibility of being an image of the sameobject as the input image. Finally, the reference images having highsimilarities are displayed (step 104).

An image matching system according to a fourth embodiment of the presentinvention will now be described in detail with reference to FIGS. 20 to22.

With reference to FIG. 20, an image matching system according to thefourth embodiment of the present invention includes an image inputsection 10, an image generation section 30, an image matching section40, a result matching section 60, a second image generation section 31,a second image matching section 41, a result display section 80, areference image storage section 70, a reference image registrationsection 75, a representative three-dimensional object model storagesection 20, a three-dimensional object model registration section 25, areference image matching result storage section 50, a reference imagematching result update section 55, a reference three-dimensional objectmodel storage section 21, and a three-dimensional object modelgeneration section 27.

The fourth embodiment has a configuration obtained by adding thereference image registration section 75, the three-dimensional objectmodel registration section 25, the reference image matching resultupdate section 55 and the three-dimensional object model generationsection to the configuration of the third embodiment.

As to components in the image matching system according to the fourthembodiment that are the same as those in the third embodiment,description will be omitted. The components added in the presentembodiment will be described.

These components nearly operate as described below. The image inputsection 10, the image generation section 30, the image matching section40, the result matching section 60, the result display section 80, thereference image storage section 70, the reference image registrationsection 75, the representative three-dimensional object model storagesection 20, the three-dimensional object model registration section 25,and the reference image matching result update section 55 conduct thesame processing as the processing conducted in the first embodiment ofthe present invention shown in FIG. 1 and the second embodiment shown inFIG. 9.

The reference three-dimensional object model storage section 21, thesecond image generation section 31, and the second image matchingsection 41 conduct the same processing as the processing conducted inthe third embodiment shown in FIG. 16.

When at the time of registration a reference image matching result isregistered in the reference image matching result storage section 50 bythe reference image matching result update section 55, thethree-dimensional object model generation section 27 generates areference three-dimensional object model associated with the referenceimage by combining representative three-dimensional object models in therepresentative three-dimensional object model storage section 20 on thebasis of information of the reference image matching result, andregisters the reference three-dimensional object model in the referencethree-dimensional object model storage section 21, or updates areference three-dimensional object model in the referencethree-dimensional object model storage section 21.

With respect to the reference images of high precedence candidates inthe matching result obtained from the result matching section 60, thesecond image generation section 31 generates comparison images close inthe input condition such as the pose and illumination condition to theinput image obtained from the image input section 10, on the basis ofeach of the reference three-dimensional object models associated withthe reference image obtained from the reference three-dimensional objectmodel storage section 21.

The second image matching section 41 compares the input image obtainedfrom the image input section 10 with each of the comparison imagesobtained from the second image generation section 31, and calculateseach similarity.

General operation of the fourth embodiment will now be described indetail with reference to FIG. 20 and flow charts shown in FIGS. 17, 21and 22.

At the time of matching of the input image, operation conducted at steps100, 101, 102 and 103 in FIG. 17 is the same as that conducted in thefirst embodiment shown in FIG. 2.

The second image generation section 31 generates comparison images thatare close in input condition such as the pose and illumination conditionto the input image obtained from the image input section, for areference image that is a high precedence candidate in the matchingresults obtained from the result matching section 60, on the basis ofeach of the reference three-dimensional object models associated withthe reference image obtained from the reference three-dimensional objectmodel storage section 21 (step 111).

The second image matching section 41 compares the input image obtainedfrom the image input section with each of the comparison images obtainedfrom the second image generation section 31, and calculates eachsimilarity (step 112). Finally, the reference image having highsimilarity is displayed (step 104).

At the time of three-dimensional object model registration, operationconducted at steps 200, 201 and 202 in FIG. 21 is the same as thatconducted in the second embodiment shown in FIG. 10. Finally, thethree-dimensional object model generation section 27 regenerates areference three-dimensional object model associated with each referenceimage by combining representative three-dimensional object models in therepresentative three-dimensional object model storage section 20 on thebasis of information of each reference image matching result in thereference image matching result storage section 50, and registers thereference three-dimensional object model in the referencethree-dimensional object model storage section 21, or replaces a storedreference three-dimensional object model by it (step 220).

At the time of reference image registration, operation conducted atsteps 210, 211 and 212 in FIG. 22 is the same as that conducted in thesecond embodiment shown in FIG. 11. Finally, the three-dimensionalobject model generation section 27 generates a referencethree-dimensional object model associated with the reference image bycombining representative three-dimensional object models in therepresentative three-dimensional object model storage section 20 on thebasis of information of the reference image matching result newlyregistered in the reference image matching result storage section 50,and additionally registers the reference three-dimensional object modelin the reference three-dimensional object model storage section 21 (step221).

Effects of the fourth embodiment which has the configuration and whichoperates as described above will now be described.

The fourth embodiment has the configuration in which the referencethree-dimensional object model generated by combining representativethree-dimensional object models is matched. Even when only one referenceimage is present every object, therefore, reference images can beretrieved by using the reference three-dimensional object models, withrespect to an input image of an object picked up under a differentcondition concerning the pose and illumination.

The present embodiment has a configuration in which reference imageshaving high similarity are extracted by using a representativethree-dimensional object model and then matching of the referencethree-dimensional object model with high precedence candidates isconducted. As a result, reference images can be retrieved at high speed.

Operation of the fourth embodiment will now be described with referenceto concrete embodiments.

In the same way as the operation of the first embodiment, representativethree-dimensional object models C_(j) (j=1, 2, . . . , N) as shown inFIG. 3 are stored in the representative three-dimensional object modelstorage section 20. Reference images R_(i) (i=1, 2, . . . , M)respectively of the objects as shown in FIG. 4 are stored in thereference image storage section 70. A result (similarity) S_(ij) ofmatching of each reference image R_(i) with the representativethree-dimensional object model C_(j) as shown in FIG. 5 is stored in thereference image matching result storage section 50.

In addition, M reference three-dimensional object models Bi (i=1, 2, . .. , M) associated with the reference image R_(i) are previously storedin the reference three-dimensional object model storage section 21 bythe processing conducted at the time of reference image registration, asshown in FIG. 18.

It is supposed that an input image I(u, v) as shown in FIG. 6 isobtained by the image input section 10 at the time of matching of theinput image (step 100 in FIG. 16). According to the same processing asthe operation in the first embodiment, R₁, R₅ and R₂ are obtained in thecited order as reference images having a high possibility of being animage of the same object as the input image as shown in FIG. 8 by theimage generation section 30, the image matching section 40, and theresult matching section 60 (steps 101, 102 and 103).

With respect to, for example, the reference images R₁, R₅ and R₂ whichare three high precedence candidates in the matching result obtainedfrom the result matching section 60, the second image generation section31 acquires associated reference three-dimensional object models B₁, B₅and B₂ from the reference three-dimensional object model storage section21, and generates comparison images H_(jk)(u, v) (j=1, 5, 2, k=1, . . ., L) which are close in input condition such as the pose andillumination to the input image obtained from the image input section 10(step 111). The second image matching section 41 finds a similarity S(I,H_(jk)) between the input image I(u, v) and each comparison imageH_(jk)(u, v), and finds a maximum similarity S_(oj)=max_(k)S(I, H_(jk))every model (step 112).

The matching results become, for example, as shown in FIG. 19, and R₅,R₁ and R₂ are obtained in the cited order as reference images having ahigh possibility of being an image of the same object as the inputimage. Finally, the reference images having high similarity aredisplayed (step 104).

At the time of three-dimensional object model registration, thethree-dimensional object model registration section 25 first registers anew representative three-dimensional object model in the representativethree-dimensional object model storage section 20. If N=50three-dimensional object models are already registered in therepresentative three-dimensional object model storage section 20, thethree-dimensional object model registration section 25 registers a newfifty-first representative three-dimensional object model C₅₁ (step 200in FIG. 21).

Subsequently, by the same processing as the operation in the secondembodiment, the reference image matching result update section 55updates the matching result S_(ij) of each reference image in thereference image matching result storage section 50 (steps 201 and 202).

Finally, the three-dimensional object model generation section 27regenerates a reference three-dimensional object model B_(i) associatedwith each reference image R_(i) (i=1, 2, . . . , M) by combiningrepresentative three-dimensional object models C_(j) in therepresentative three-dimensional object model storage section 20 on thebasis of information of each reference image matching result S_(ij) inthe reference image matching result storage section 50, and replaces thereference three-dimensional object model already stored in the referencethree-dimensional object model storage section 21 by the referencethree-dimensional object model B_(i) (step 220).

Denoting the shape and texture of the representative three-dimensionalobject model C_(j) (j=1, 2, . . . , N) respectively by P_(Qj)(x, y, z)and T_(Qj)(R, G, B) and denoting the shape and texture of the referencethree-dimensional object model B_(i) (i=1, 2, . . . , M) respectively byP_(Qi)(x, y, z) and T_(Qi)(R, G, B), the reference three-dimensionalobject model is calculated according to, for example, the followingexpressions.P _(Qi)(x,y,z)=Σ_(j) f(S _(ij))P _(Qj)(x,y,z)T _(Qi)(R,G,B)=Σ_(j) f(S _(ij))T _(Qj)(R,G,B)

Here, f is a function that monotonously increases as S_(ij) increases,and that satisfies the relation Σ_(j)f(S_(ij))=1. As the simplestexample, f can be implemented by f(S_(ij))=S_(ij)/Σ_(j)S_(ij).

At the time of reference image registration, the reference imageregistration section 75 first registers a new reference image in thereference image storage section 70. If M=100 reference images arealready registered in the reference image storage section 70, thereference image registration section 75 registers a new hundred firstreference image R₁₀₁ in the reference image storage section 70 (step 210in FIG. 22).

Subsequently, the reference image matching result update section 55 addsa matching result S_(101,j) associated with the reference image R₁₀₁ tothe reference image matching result storage section 50 by conducting thesame processing as the operation in the second embodiment (steps 211 and212).

Finally, the three-dimensional object model generation section 27generates a reference three-dimensional object model B₁₀₁ associatedwith the reference image R₁₀₁ by combining representativethree-dimensional object models C_(j) in the representativethree-dimensional object model storage section 20 on the basis ofinformation of the reference image matching result S_(101,j) in thereference image matching result storage section 50, and adds thereference three-dimensional object model B₁₀₁ to the referencethree-dimensional object model storage section 21 (step 221).

An image matching system according to a fifth embodiment of the presentinvention will now be described in detail with reference to FIGS. 23 and24.

With reference to FIG. 23, an image matching system according to thefifth embodiment of the present invention includes an image inputsection 10, an image generation section 30, an image matching section40, a result matching section 60, an image conversion section 36, apartial image matching section 45, a result display section 80, areference image storage section 70, a representative three-dimensionalobject model storage section 20, a reference image matching resultstorage section 50, and a reference three-dimensional object modelstorage section 21.

These components nearly operate as described below. The image inputsection 10, the image generation section 30, the image matching section40, the result matching section 60, the result display section 80, thereference image storage section 70, and the representativethree-dimensional object model storage section 20 conduct the sameprocessing as the processing conducted in the first embodiment of thepresent invention shown in FIG. 1.

With respect to a reference image of a high precedence candidate in thematching result obtained from the result obtained from the resultmatching section 60, the image conversion section 36 converts the inputimage and/or the reference image so as to make the input condition (suchas the pose condition) the same on the basis of each referencethree-dimensional object model associated with the reference imageobtained from the reference three-dimensional object model storagesection 21. Thus, the image conversion section 36 generates partialimages. The image conversion section 36 is similar to the imageconversion section 35 in the second conventional technique shown in FIG.28.

The partial image matching section 45 conducts comparison on the partialimages of the converted input image and reference image obtained fromthe image conversion section 36, and calculates the similarity. Thesimilarity calculation is conducted in the same way as the step 102.

General operation of the fifth embodiment will now be described indetail with reference to FIG. 23 and a flow chart shown in FIG. 24.

At the time of input image matching, steps 100, 101, 102 and 103 shownin FIG. 24 are the same as the operation conducted in the firstembodiment shown in FIG. 2. With respect to a reference image of a highprecedence candidate in the matching result obtained from the resultobtained from the result matching section 60, the image conversionsection 36 converts the input image and/or the reference image so as tomake the input condition (such as the pose condition) the same on thebasis of each reference three-dimensional object model associated withthe reference image obtained from the reference three-dimensional objectmodel storage section 21. Thus, the image conversion section 36generates partial images (step 121).

The partial image matching section 45 conducts comparison on the partialimages of the converted input image and reference image obtained fromthe image conversion section 36, and calculates the similarity (step122). Finally, the reference image having high similarity is displayed(step 104).

In the fifth embodiment of the present invention, the image conversionsection 36 converts the input image and/or the reference image.Alternatively, the reference image may be previously converted to thatunder a standard input condition (for example, a standard posecondition) and stored, and the image conversion section 36 may convertthe input image to that under the standard input condition (for example,the standard pose condition). By doing so, it becomes unnecessary toconvert the reference image each time the matching is conducted, and thematching time can be shortened.

An image matching system according to a sixth embodiment of the presentinvention will now be described in detail with reference to FIG. 25.

With reference to FIG. 25, an image matching system according to thesixth embodiment of the present invention includes an image inputsection 10, an image generation section 30, an image matching section40, a result matching section 60, an image conversion section 36, apartial image matching section 45, a result display section 80, areference image storage section 70, a reference image registrationsection 75, a representative three-dimensional object model storagesection 20, a three-dimensional object model registration section 25, areference image matching result storage section 50, a reference imagematching result update section 55, a reference three-dimensional objectmodel storage section 21, and a three-dimensional object modelgeneration section 27.

These components nearly operate as described below. The image inputsection 10, the image generation section 30, the image matching section40, the result matching section 60, the result display section 80, thereference image storage section 70, the reference image registrationsection 75, the representative three-dimensional object model storagesection 20, the three-dimensional object model registration section 25,and the reference image matching result update section 55 conduct thesame processing as the processing conducted in the first embodiment ofthe present invention shown in FIG. 1 and the second embodiment shown inFIG. 9.

The reference three-dimensional object model storage section 21 and thethree-dimensional object model generation section 27 conduct the sameprocessing as the processing conducted in the third embodiment shown inFIG. 16 and the fourth embodiment of the present invention shown in FIG.20.

With respect to a reference image of a high precedence candidate in thematching result obtained from the result obtained from the resultmatching section 60, the image conversion section 36 converts the inputimage and/or the reference image so as to make the input condition (suchas the pose condition) the same on the basis of each referencethree-dimensional object model associated with the reference imageobtained from the reference three-dimensional object model storagesection 21. Thus, the image conversion section 36 generates partialimages. The partial image matching section 45 conducts comparison on thepartial images of the converted input image and reference image obtainedfrom the image conversion section 36, and calculates the similarity.

General operation of the sixth embodiment will now be described indetail with reference to FIG. 25 and the flow chart shown in FIG. 24.

At the time of input image matching, steps 100, 101, 102 and 103 shownin FIG. 24 are the same as the operation conducted in the firstembodiment shown in FIG. 2. With respect to a reference image of a highprecedence candidate in the matching result obtained from the resultobtained from the result matching section 60, the image conversionsection 36 converts the input image and/or the reference image so as tomake the input condition (such as the pose condition) the same on thebasis of each reference three-dimensional object model associated withthe reference image obtained from the reference three-dimensional objectmodel storage section 21. Thus, the image conversion section 36generates partial images (step 121).

The partial image matching section 45 conducts comparison on the partialimages of the converted input image and reference image obtained fromthe image conversion section 36, and calculates the similarity (step122). Finally, the reference image having high similarity is displayed(step 104).

In the sixth embodiment of the present invention, the image conversionsection 36 converts the input image and/or the reference image.Alternatively, the reference image may be previously converted to thatunder a standard input condition (for example, a standard posecondition) and stored, and the image conversion section 36 may convertthe input image to that under the standard input condition (for example,the standard pose condition).

When finding the similarity S(I, G_(jk)) between the input image I(u, v)and each of the comparison images G_(jk)(u, v) in the first to sixthembodiments of the present invention, the image matching section 40finds one similarity S(I, G_(jk)) on the whole. Alternatively, the imagematching section 40 may find similarity S(I, G′_(jkm)) every partialregion m, and find a maximum similarity S′_(0jm)=max_(k)S(I, G′_(jkm))every model.

The partial regions are, for example, regions as shown in FIG. 29. Inthis case, similarity S′_(ijm)=max_(k)S(R_(i), G′_(jkm)) of each partialregion m is stored in the reference image matching result storagesection 50 as well. The result matching section 60 calculates asimilarity D_(i)=Σ_(m)D(S′_(0jm), S′_(ijm)) between the matching resultS′_(0jm) and the matching result S′_(ijm) of each of the referenceimages in the reference image matching result storage section 50, andextracts reference images in the descending order of the similarityD_(i) of the matching result. Furthermore, in the three-dimensionalobject model generation section 27 in the fourth and sixth embodimentsas well, representative three-dimensional object models may be combinedevery partial region.

In the first to sixth embodiments of the present invention, operation ofretrieving an image of the same object as the input image from among alarge number of reference images has been described. However, it is alsopossible to apply the present invention to one-to-one matching fordetermining whether a specific reference image is an image of the sameobject as the input image.

It is supposed that a specific reference image is R₁. In the first andsecond embodiments, the result matching section 60 calculates asimilarity D₁=D(S_(0j), S_(1j)) between the matching result S_(0j) ofthe input image and the matching result S_(1j) of the reference imageR₁. If the similarity D₁ is greater than a threshold, R₁ can be judgedto be the same object as the input image. In the third, fourth, fifthand sixth embodiments, the judgment can be made by determining whetherthe similarity between the input image and a specific reference image inthe second image matching section 41 and the partial image matchingsection 45 is greater than a threshold.

In the image matching system according to the present invention, it is amatter of course that functions of components can be implemented byusing hardware. The functions can also be implemented by reading animage matching program 500 into a computer to make the computer executethe functions of components. The image matching program 500 is stored ina recording medium such as a magnetic disk or a semiconductor memory.The computer reads the image matching program 500 from the recordingmedium.

INDUSTRIAL APPLICABILITY

The present invention can be used for person identification, individualauthentication, or the like using an image of a face or the like.

1. An image matching system for retrieving a reference image similar toan input image, the image matching system comprising: means for making amatch between the input image and a plurality of representativethree-dimensional object models; means for making a match between thereference image and the representative three-dimensional object models;and means for retrieving the reference image similar to the input imageby using a result of the match between the input image and therepresentative three-dimensional object models and a result of the matchbetween the reference image and the representative three-dimensionalobject models.
 2. The image matching system according to claim 1,further comprising: means for finding a reference three-dimensionalobject model associated with the reference image similar to the inputimage; and means for newly retrieving the reference image similar to theinput image by using the reference three-dimensional object model andthe input image.
 3. The image matching system according to claim 1,further comprising: means for finding a reference three-dimensionalobject model associated with the reference image similar to the inputimage; conversion means for squaring an input condition of the inputimage with that of the reference image by converting the input imageand/or the reference image on the basis of the referencethree-dimensional object model; and means for retrieving the referenceimage associated with the input image by making a match between theinput image and reference image squared in input condition.
 4. The imagematching system according to claim 3, wherein the conversion meanspreviously converts the reference image, and squares an input conditionof the input image with that of the reference image.
 5. The imagematching system according to claim 1, comprising: image input means forinputting the input image; a representative three-dimensional objectmodel storage section for storing a plurality of representativethree-dimensional object models; image generation means for generatingat least one comparison image close in input condition to the inputimage every representative three-dimensional object model on the basisof the representative three-dimensional object models stored in therepresentative three-dimensional object model storage section; imagematching means for calculating a similarity between the input image andeach of the comparison images generated by the image generation means,selecting a maximum similarity with respect to comparison imagesassociated with each representative three-dimensional object model, andregarding the maximum similarity as a similarity between the input imageand the representative three-dimensional object model; a reference imagestorage section for storing the reference images of objects; a referenceimage matching result storage section for storing similarities betweenthe reference images stored in the reference image storage section andrepresentative three-dimensional object models stored in therepresentative three-dimensional object model storage section; andresult matching means for extracting the reference image similar to theinput image on the basis of similarities between the input image and therepresentative three-dimensional object models calculated by the imagematching means and similarities between the reference images and therepresentative three-dimensional object models stored in the referenceimage matching result storage section.
 6. The image matching systemaccording to claim 5, further comprising: three-dimensional object modelregistration means for registering representative three-dimensionalobject models in the representative three-dimensional object modelstorage section; reference image registration means for registeringreference images in the reference image storage section; and referenceimage matching result update means for conducting calculation of thesimilarities using the image matching means, on a combination of areference image and a representative three-dimensional object modelnewly generated by registration, when a new representativethree-dimensional object model is registered in the representativethree-dimensional object model storage section by the three-dimensionalobject model registration means, or when a new reference image isregistered in the reference image storage section by the reference imageregistration means, and adding a result of the calculation to results inthe reference image matching result storage section.
 7. The imagematching system according to claim 5, wherein the image matching meanscalculates a similarity between the input image and a representativethree-dimensional object model every partial region, the reference imagematching result storage section stores similarities between thereference images stored in the reference image storage section andrepresentative three-dimensional object models stored in therepresentative three-dimensional object model storage section, everypartial region, and the result matching means extracts the referenceimage similar to the input image on the basis of similarities betweenthe input image and the representative three-dimensional object modelscalculated by the image matching means every partial region andsimilarities between the reference images and the representativethree-dimensional object models, stored in the reference image matchingresult storage section every partial region.
 8. The image matchingsystem according to claim 5, wherein the result matching meanscalculates similarities between similarities between the input image andthe representative three-dimensional object models and similaritiesbetween the reference images and the representative three-dimensionalobject models, and in the calculation provides the resultantsimilarities with weights on the basis of candidate precedence ofsimilarities between the input image and the comparison images and therepresentative three-dimensional object models.
 9. The image matchingsystem according to claim 2, comprising: image input means for inputtingthe input image; a representative three-dimensional object model storagesection for storing a plurality of representative three-dimensionalobject models; image generation means for generating at least onecomparison image close in input condition to the input image everyrepresentative three-dimensional object model on the basis of therepresentative three-dimensional object models stored in therepresentative three-dimensional object model storage section; imagematching means for calculating a similarity between the input image andeach of the comparison images generated by the image generation means,selecting a maximum similarity with respect to comparison imagesassociated with each representative three-dimensional object model, andregarding the maximum similarity as a similarity between the input imageand the representative three-dimensional object model; a reference imagestorage section for storing the reference images of objects; a referenceimage matching result storage section for storing similarities betweenthe reference images stored in the reference image storage section andrepresentative three-dimensional object models stored in therepresentative three-dimensional object model storage section; resultmatching means for extracting the reference image similar to the inputimage on the basis of similarities between the input image and therepresentative three-dimensional object models calculated by the imagematching means and similarities between the reference images and therepresentative three-dimensional object models stored in the referenceimage matching result storage section; a reference three-dimensionalobject model storage section for storing reference three-dimensionalobject models associated with the reference images stored in thereference image storage section; second image generation means forobtaining reference three-dimensional object models associated withreference images extracted from the result matching means, from thereference three-dimensional object model storage section, and generatingat least one second comparison image close in input condition to theinput image every reference three-dimensional object model on the basisof the obtained reference three-dimensional object models; and secondimage matching means for calculating similarities between the inputimage and second comparison images generated by the second imagegeneration means, selecting a maximum similarity from among secondcomparison images associated with each of the referencethree-dimensional object models, and regarding the maximum similarity asa similarity between the input image and the reference three-dimensionalobject model.
 10. The image matching system according to claim 9,further comprising: three-dimensional object model registration meansfor registering representative three-dimensional object models in therepresentative three-dimensional object model storage section; referenceimage registration means for registering reference images in thereference image storage section; and reference image matching resultupdate means for conducting calculation of the similarities using theimage matching means, on a combination of a reference image and arepresentative three-dimensional object model newly generated byregistration, when a new representative three-dimensional object modelis registered in the representative three-dimensional object modelstorage section by the three-dimensional object model registrationmeans, or when a new reference image is registered in the referenceimage storage section by the reference image registration means, andadding a result of the calculation to results in the reference imagematching result storage section; and three-dimensional object modelgeneration means responsive to registration of a similarity between thereference image and the representative three-dimensional object model inthe reference image matching result storage section conducted by thereference image matching result update means, for generating thereference three-dimensional object model associated with the referenceimage by combining the representative three-dimensional object modelsstored in the representative three-dimensional object model storagesection on the basis of the similarity, and registering the generatedreference three-dimensional object model in the referencethree-dimensional object model storage section.
 11. The image matchingsystem according to claim 10, wherein the three-dimensional object modelgeneration means generates a reference three-dimensional object modelassociated with each reference image by combining representativethree-dimensional object models stored in the representativethree-dimensional object model storage section every partial region, onthe basis of similarities obtained every partial region betweenreference images stored in the reference image storage section andrepresentative three-dimensional object models stored in therepresentative three-dimensional object model storage section, andregisters the generated reference three-dimensional object model in thereference three-dimensional object model storage section.
 12. The imagematching system according to claim 9, wherein the image matching meanscalculates a similarity between the input image and a representativethree-dimensional object model every partial region, the reference imagematching result storage section stores similarities between thereference images stored in the reference image storage section andrepresentative three-dimensional object models stored in therepresentative three-dimensional object model storage section, everypartial region, and the result matching means extracts the referenceimage similar to the input image on the basis of similarities betweenthe input image and the representative three-dimensional object modelscalculated by the image matching means every partial region andsimilarities between the reference images and the representativethree-dimensional object models, stored in the reference image matchingresult storage section every partial region.
 13. The image matchingsystem according to claim 9, wherein the result matching meanscalculates similarities between similarities between the input image andthe representative three-dimensional object models and similaritiesbetween the reference images and the representative three-dimensionalobject models, and in the calculation, provides the resultantsimilarities with weights on the basis of candidate precedence ofsimilarities between the input image and the comparison images and therepresentative three-dimensional object models.
 14. The image matchingsystem according to claim 3, comprising: image input means for inputtingthe input image; a representative three-dimensional object model storagesection for storing a plurality of representative three-dimensionalobject models; image generation means for generating at least onecomparison image close in input condition to the input image everyrepresentative three-dimensional object model on the basis of therepresentative three-dimensional object models stored in therepresentative three-dimensional object model storage section; imagematching means for calculating a similarity between the input image andeach of the comparison images generated by the image generation means,selecting a maximum similarity with respect to comparison imagesassociated with each representative three-dimensional object model, andregarding the maximum similarity as a similarity between the input imageand the representative three-dimensional object model; a reference imagestorage section for storing the reference images of objects; a referenceimage matching result storage section for storing similarities betweenthe reference images stored in the reference image storage section andrepresentative three-dimensional object models stored in therepresentative three-dimensional object model storage section; resultmatching means for extracting the reference image similar to the inputimage on the basis of similarities between the input image and therepresentative three-dimensional object models calculated by the imagematching means and similarities between the reference images and therepresentative three-dimensional object models stored in the referenceimage matching result storage section; a reference three-dimensionalobject model storage section for storing reference three-dimensionalobject models associated with the reference images stored in thereference image storage section; image conversion means for obtainingreference three-dimensional object models associated with referenceimages extracted from the result matching means, from the referencethree-dimensional object model storage section, squaring an inputcondition of the input image with that of the reference image extractedby the result matching means by converting the input image and/or thereference image extracted by the result matching means, on the basis ofthe obtained reference three-dimensional object models, and generatingpartial images respectively of the input image and the reference imagesquared in input condition with each other; and partial image matchingmeans for calculating a similarity between the partial image of theinput image and the partial image of the reference image generated bythe image conversion means.
 15. The image matching system according toclaim 14, further comprising: three-dimensional object modelregistration means for registering representative three-dimensionalobject models in the representative three-dimensional object modelstorage section; reference image registration means for registeringreference images in the reference image storage section; and referenceimage matching result update means for conducting calculation of thesimilarities using the image matching means, on a combination of areference image and a representative three-dimensional object modelnewly generated by registration, when a new representativethree-dimensional object model is registered in the representativethree-dimensional object model storage section by the three-dimensionalobject model registration means, or when a reference image is registeredin the reference image storage section by the reference imageregistration means, and adding a result of the calculation to results inthe reference image matching result storage section; andthree-dimensional object model generation means responsive toregistration of a similarity between the reference image and therepresentative three-dimensional object model in the reference imagematching result storage section conducted by the reference imagematching result update means, for generating the referencethree-dimensional object model associated with the reference image bycombining the representative three-dimensional object models stored inthe representative three-dimensional object model storage section on thebasis of the similarity, and registering the generated referencethree-dimensional object model in the reference three-dimensional objectmodel storage section.
 16. The image matching system according to claim15, wherein the three-dimensional object model generation meansgenerates a reference three-dimensional object model associated witheach reference image by combining representative three-dimensionalobject models stored in the representative three-dimensional objectmodel storage section every partial region, on the basis of similaritiesobtained every partial region between reference images stored in thereference image storage section and representative three-dimensionalobject models stored in the representative three-dimensional objectmodel storage section, and registers the generated referencethree-dimensional object model in the reference three-dimensional objectmodel storage section.
 17. The image matching system according to claim14, wherein the image matching means calculates a similarity between theinput image and a representative three-dimensional object model everypartial region, the reference image matching result storage sectionstores similarities between the reference images stored in the referenceimage storage section and representative three-dimensional object modelsstored in the representative three-dimensional object model storagesection, every partial region, and the result matching means extractsthe reference image similar to the input image on the basis ofsimilarities between the input image and the representativethree-dimensional object models calculated by the image matching meansevery partial region and similarities between the reference images andthe representative three-dimensional object models, stored in thereference image matching result storage section every partial region.18. The image matching system according to claim 14, wherein the resultmatching means calculates similarities between similarities between theinput image and the representative three-dimensional object models andsimilarities between the reference images and the representativethree-dimensional object models, and in the calculation, provides theresultant similarities with weights on the basis of candidate precedenceof similarities between the input image and the comparison images andthe representative three-dimensional object models.
 19. The imagematching system according to claim 1, wherein the object is a humanface.
 20. An image matching method for retrieving a reference imagesimilar to an input image, the image matching method comprising: a stepof making a match between the input image and a plurality ofrepresentative three-dimensional object models; a step of making a matchbetween the reference image and the representative three-dimensionalobject models; and a step of retrieving the reference image similar tothe input image by using a result of the match between the input imageand the representative three-dimensional object models and a result ofthe match between the reference image and the representativethree-dimensional object models.
 21. The image matching method accordingto claim 20, further comprising: a step of finding a referencethree-dimensional object model associated with the reference imagesimilar to the input image; and a step of newly retrieving the referenceimage similar to the input image by using the referencethree-dimensional object model and the input image.
 22. The imagematching method according to claim 20, further comprising the steps of:a step of finding a reference three-dimensional object model associatedwith the reference image similar to the input image; a conversion stepof squaring an input condition of the input image with that of thereference image by converting the input image and/or the reference imageon the basis of the reference three-dimensional object model; and a stepof retrieving the reference image associated with the input image bymaking a match between the input image and reference image squared ininput condition.
 23. The image matching method according to claim 22,wherein at the conversion step, the reference image is previouslyconverted, and an input condition of the input image is squared withthat of the reference image.
 24. The image matching method according toclaim 20, comprising: an image input step of inputting the input image;a step of storing a plurality of representative three-dimensional objectmodels in a representative three-dimensional object model storagesection; an image generation step of generating at least one comparisonimage close in input condition to the input image every representativethree-dimensional object model on the basis of the representativethree-dimensional object models stored in the representativethree-dimensional object model storage section; an image matching stepof calculating a similarity between the input image and each of thecomparison images generated by the image generation means, selecting amaximum similarity with respect to comparison images associated witheach representative three-dimensional object model, and regarding themaximum similarity as a similarity between the input image and therepresentative three-dimensional object model; a step of storing thereference images of objects in a reference image storage section; a stepof storing similarities between the reference images stored in thereference image storage section and representative three-dimensionalobject models stored in the representative three-dimensional objectmodel storage section, in a reference image matching result storagesection; and a result matching step of extracting the reference imagesimilar to the input image on the basis of similarities between theinput image and the representative three-dimensional object modelscalculated by the image matching means and similarities between thereference images and the representative three-dimensional object modelsstored in the reference image matching result storage section.
 25. Theimage matching method according to claim 24, further comprising: athree-dimensional object model registration step of registeringrepresentative three-dimensional object models in the representativethree-dimensional object model storage section; a reference imageregistration step of registering reference images in the reference imagestorage section; and a reference image matching result update step ofconducting calculation of the similarities using the image matchingmeans, on a combination of a reference image and a representativethree-dimensional object model newly generated by registration, when anew representative three-dimensional object model is registered in therepresentative three-dimensional object model storage section by thethree-dimensional object model registration means, or when a newreference image is registered in the reference image storage section bythe reference image registration means, and adding a result of thecalculation to results in the reference image matching result storagesection.
 26. The image matching method according to claim 24, wherein atthe image matching step, a similarity between the input image and arepresentative three-dimensional object model is calculated everypartial region, the reference image matching result storage sectionstores similarities between the reference images stored in the referenceimage storage section and representative three-dimensional object modelsstored in the representative three-dimensional object model storagesection, every partial region, and at the result matching step, thereference image similar to the input image is extracted on the basis ofsimilarities between the input image and the representativethree-dimensional object models calculated by the image matching meansevery partial region and similarities between the reference images andthe representative three-dimensional object models, stored in thereference image matching result storage section every partial region.27. The image matching method according to claim 24, wherein at theresult matching step, similarities between similarities between theinput image and the representative three-dimensional object models andsimilarities between the reference images and the representativethree-dimensional object models are calculated, and in the calculationthe resultant similarities are provided with weights on the basis ofcandidate precedence of similarities between the input image and thecomparison images and the representative three-dimensional objectmodels.
 28. The image matching method according to claim 21, comprising:an image input step of inputting the input image; a step of storing aplurality of representative three-dimensional object models in arepresentative three-dimensional object model storage section; an imagegeneration step of generating at least one comparison image close ininput condition to the input image every representativethree-dimensional object model on the basis of the representativethree-dimensional object models stored in the representativethree-dimensional object model storage section; an image matching stepof calculating a similarity between the input image and each of thecomparison images generated at the image generation step, selecting amaximum similarity with respect to comparison images associated witheach representative three-dimensional object model, and regarding themaximum similarity as a similarity between the input image and therepresentative three-dimensional object model; a step of storing thereference images of objects in a reference image storage section; a stepof storing similarities between the reference images stored in thereference image storage section and representative three-dimensionalobject models stored in the representative three-dimensional objectmodel storage section, in a reference image matching result storagesection; a result matching step of extracting the reference imagesimilar to the input image on the basis of similarities between theinput image and the representative three-dimensional object modelscalculated at the image matching step and similarities between thereference images and the representative three-dimensional object modelsstored in the reference image matching result storage section; a step ofstoring reference three-dimensional object models associated with thereference images stored in the reference image storage section, in areference three-dimensional object model storage section; a second imagegeneration step of obtaining reference three-dimensional object modelsassociated with reference images extracted at the result matching step,from the reference three-dimensional object model storage section, andgenerating at least one second comparison image close in input conditionto the input image every reference three-dimensional object model on thebasis of the obtained reference three-dimensional object models; and asecond image matching step of calculating similarities between the inputimage and second comparison images generated at the second imagegeneration step, selecting a maximum similarity from among secondcomparison images associated with each of the referencethree-dimensional object models, and regarding the maximum similarity asa similarity between the input image and the reference three-dimensionalobject model.
 29. The image matching method according to claim 28,further comprising: a three-dimensional object model registration stepof registering representative three-dimensional object models in therepresentative three-dimensional object model storage section; areference image registration step of registering reference images in thereference image storage section; and a reference image matching resultupdate step of conducting calculation of the similarities using theimage matching means, on a combination of a reference image and arepresentative three-dimensional object model newly generated byregistration, when a new representative three-dimensional object modelis registered in the representative three-dimensional object modelstorage section at the three-dimensional object model registration step,or when a new reference image is registered in the reference imagestorage section at the reference image registration step, and adding aresult of the calculation to results in the reference image matchingresult storage section; and a three-dimensional object model generationstep of, in response to registration of a similarity between thereference image and the representative three-dimensional object model inthe reference image matching result storage section conducted at thereference image matching result update step, generating the referencethree-dimensional object model associated with the reference image bycombining the representative three-dimensional object models stored inthe representative three-dimensional object model storage section on thebasis of the similarity, and registering the generated referencethree-dimensional object model in the reference three-dimensional objectmodel storage section.
 30. The image matching method according to claim29, wherein at the three-dimensional object model generation step, areference three-dimensional object model associated with each referenceimage is generated by combining representative three-dimensional objectmodels stored in the representative three-dimensional object modelstorage section every partial region, on the basis of similaritiesobtained every partial region between reference images stored in thereference image storage section and representative three-dimensionalobject models stored in the representative three-dimensional objectmodel storage section, and the generated reference three-dimensionalobject model is registered in the reference three-dimensional objectmodel storage section.
 31. The image matching method according to claim28, wherein at the image matching step, a similarity between the inputimage and a representative three-dimensional object model is calculatedevery partial region, the reference image matching result storagesection stores similarities between the reference images stored in thereference image storage section and representative three-dimensionalobject models stored in the representative three-dimensional objectmodel storage section, every partial region, and at the result matchingstep, the reference image similar to the input image is extracted on thebasis of similarities between the input image and the representativethree-dimensional object models calculated by the image matching meansevery partial region and similarities between the reference images andthe representative three-dimensional object models, stored in thereference image matching result storage section every partial region.32. The image matching method according to claim 28, wherein at theresult matching step, similarities between similarities between theinput image and the representative three-dimensional object models andsimilarities between the reference images and the representativethree-dimensional object models are calculated, and in the calculation,the resultant similarities are provided with weights on the basis ofcandidate precedence of similarities between the input image and thecomparison images and the representative three-dimensional objectmodels.
 33. The image matching method according to claim 22, comprising:an image input step of inputting the input image; a step of storing aplurality of representative three-dimensional object models in arepresentative three-dimensional object model storage section; an imagegeneration step of generating at least one comparison image close ininput condition to the input image every representativethree-dimensional object model on the basis of the representativethree-dimensional object models stored in the representativethree-dimensional object model storage section; an image matching stepof calculating a similarity between the input image and each of thecomparison images generated at the image generation means, selecting amaximum similarity with respect to comparison images associated witheach representative three-dimensional object model, and regarding themaximum similarity as a similarity between the input image and therepresentative three-dimensional object model; a step of storing thereference images of objects in a reference image storage section; a stepof storing similarities between the reference images stored in thereference image storage section and representative three-dimensionalobject models stored in the representative three-dimensional objectmodel storage section, in a reference image matching result storagesection; a result matching step of extracting the reference imagesimilar to the input image on the basis of similarities between theinput image and the representative three-dimensional object modelscalculated at the image matching step and similarities between thereference images and the representative three-dimensional object modelsstored in the reference image matching result storage section; a step ofstoring reference three-dimensional object models associated with thereference images stored in the reference image storage section, in areference three-dimensional object model storage section; an imageconversion step of obtaining reference three-dimensional object modelsassociated with reference images extracted at the result matching step,from the reference three-dimensional object model storage section,squaring an input condition of the input image with that of thereference image extracted at the result matching step by converting theinput image and/or the reference image extracted at the result matchingstep, on the basis of the obtained reference three-dimensional objectmodels, and generating partial images respectively of the input imageand the reference image squared in input condition with each other; anda partial image matching step of calculating a similarity between thepartial image of the input image and the partial image of the referenceimage generated at the image conversion step.
 34. The image matchingmethod according to claim 33, further comprising: a three-dimensionalobject model registration step of registering representativethree-dimensional object models in the representative three-dimensionalobject model storage section; a reference image registration step ofregistering reference images in the reference image storage section; anda reference image matching result update step of conducting calculationof the similarities at the image matching step, on a combination of areference image and a representative three-dimensional object modelnewly generated by registration, when a new representativethree-dimensional object model is registered in the representativethree-dimensional object model storage section at the three-dimensionalobject model registration step, or when a reference image is registeredin the reference image storage section at the reference imageregistration step, and adding a result of the calculation to results inthe reference image matching result storage section; and athree-dimensional object model generation step of, in response toregistration of a similarity between the reference image and therepresentative three-dimensional object model in the reference imagematching result storage section conducted at the reference imagematching result update step, for generating the referencethree-dimensional object model associated with the reference image bycombining the representative three-dimensional object models stored inthe representative three-dimensional object model storage section on thebasis of the similarity, and registering the generated referencethree-dimensional object model in the reference three-dimensional objectmodel storage section.
 35. The image matching method according to claim34, wherein at the three-dimensional object model generation step, areference three-dimensional object model associated with each referenceimage is generated by combining representative three-dimensional objectmodels stored in the representative three-dimensional object modelstorage section every partial region, on the basis of similaritiesobtained every partial region between reference images stored in thereference image storage section and representative three-dimensionalobject models stored in the representative three-dimensional objectmodel storage section, and the generated reference three-dimensionalobject model is registered in the reference three-dimensional objectmodel storage section.
 36. The image matching method according to claim33, wherein at the image matching step, a similarity between the inputimage and a representative three-dimensional object model is calculatedevery partial region, the reference image matching result storagesection stores similarities between the reference images stored in thereference image storage section and representative three-dimensionalobject models stored in the representative three-dimensional objectmodel storage section, every partial region, and at the result matchingstep, the reference image similar to the input image is extracted on thebasis of similarities between the input image and the representativethree-dimensional object models calculated at the image matching stepevery partial region and similarities between the reference images andthe representative three-dimensional object models, stored in thereference image matching result storage section every partial region.37. The image matching method according to claim 33, wherein at theresult matching step, similarities between similarities between theinput image and the representative three-dimensional object models andsimilarities between the reference images and the representativethree-dimensional object models are calculated, and in the calculation,the resultant similarities are provided with weights on the basis ofcandidate precedence of similarities between the input image and thecomparison images and the representative three-dimensional objectmodels.
 38. The image matching method according to claim 20, wherein theobject is a human face.
 39. A program for making a computer execute animage matching method to retrieve a reference image similar to an inputimage, the image matching method comprising: a step of making a matchbetween the input image and a plurality of representativethree-dimensional object models; a step of making a match between thereference image and the representative three-dimensional object models;and a step of retrieving the reference image similar to the input imageby using a result of the match between the input image and therepresentative three-dimensional object models and a result of the matchbetween the reference image and the representative three-dimensionalobject models.
 40. The image matching program according to claim 39, theimage matching method further comprising: a step of finding a referencethree-dimensional object model associated with the reference imagesimilar to the input image; and a step of newly retrieving the referenceimage similar to the input image by using the referencethree-dimensional object model and the input image.
 41. The imagematching program according to claim 39, the image matching methodfurther comprising the steps of: a step of finding a referencethree-dimensional object model associated with the reference imagesimilar to the input image; a conversion step of squaring an inputcondition of the input image with that of the reference image byconverting the input image and/or the reference image on the basis ofthe reference three-dimensional object model; and a step of retrievingthe reference image associated with the input image by making a matchbetween the input image and reference image squared in input condition.42. The image matching program according to claim 41, wherein at theconversion step, the reference image is previously converted, and aninput condition of the input image is squared with that of the referenceimage.
 43. The image matching program according to claim 39, the imagematching method comprising: an image input step of inputting the inputimage; a step of storing a plurality of representative three-dimensionalobject models in a representative three-dimensional object model storagesection; an image generation step of generating at least one comparisonimage close in input condition to the input image every representativethree-dimensional object model on the basis of the representativethree-dimensional object models stored in the representativethree-dimensional object model storage section; an image matching stepof calculating a similarity between the input image and each of thecomparison images generated by the image generation means, selecting amaximum similarity with respect to comparison images associated witheach representative three-dimensional object model, and regarding themaximum similarity as a similarity between the input image and therepresentative three-dimensional object model; a step of storing thereference images of objects in a reference image storage section; a stepof storing similarities between the reference images stored in thereference image storage section and representative three-dimensionalobject models stored in the representative three-dimensional objectmodel storage section, in a reference image matching result storagesection; and a result matching step of extracting the reference imagesimilar to the input image on the basis of similarities between theinput image and the representative three-dimensional object modelscalculated by the image matching means and similarities between thereference images and the representative three-dimensional object modelsstored in the reference image matching result storage section.
 44. Theimage matching program according to claim 43, the image matching methodfurther comprising: a three-dimensional object model registration stepof registering representative three-dimensional object models in therepresentative three-dimensional object model storage section; areference image registration step of registering reference images in thereference image storage section; and a reference image matching resultupdate step of conducting calculation of the similarities using theimage matching means, on a combination of a reference image and arepresentative three-dimensional object model newly generated byregistration, when a new representative three-dimensional object modelis registered in the representative three-dimensional object modelstorage section by the three-dimensional object model registrationmeans, or when a new reference image is registered in the referenceimage storage section by the reference image registration means, andadding a result of the calculation to results in the reference imagematching result storage section.
 45. The image matching programaccording to claim 43, wherein at the image matching step, a similaritybetween the input image and a representative three-dimensional objectmodel is calculated every partial region, the reference image matchingresult storage section stores similarities between the reference imagesstored in the reference image storage section and representativethree-dimensional object models stored in the representativethree-dimensional object model storage section, every partial region,and at the result matching step, the reference image similar to theinput image is extracted on the basis of similarities between the inputimage and the representative three-dimensional object models calculatedby the image matching means every partial region and similaritiesbetween the reference images and the representative three-dimensionalobject models, stored in the reference image matching result storagesection every partial region.
 46. The image matching program accordingto claim 43, wherein at the result matching step, similarities betweensimilarities between the input image and the representativethree-dimensional object models and similarities between the referenceimages and the representative three-dimensional object models arecalculated, and in the calculation the resultant similarities areprovided with weights on the basis of candidate precedence ofsimilarities between the input image and the comparison images and therepresentative three-dimensional object models.
 47. The image matchingprogram according to claim 40, the image matching method comprising: animage input step of inputting the input image; a step of storing aplurality of representative three-dimensional object models in arepresentative three-dimensional object model storage section; an imagegeneration step of generating at least one comparison image close ininput condition to the input image every representativethree-dimensional object model on the basis of the representativethree-dimensional object models stored in the representativethree-dimensional object model storage section; an image matching stepof calculating a similarity between the input image and each of thecomparison images generated at the image generation step, selecting amaximum similarity with respect to comparison images associated witheach representative three-dimensional object model, and regarding themaximum similarity as a similarity between the input image and therepresentative three-dimensional object model; a step of storing thereference images of objects in a reference image storage section; a stepof storing similarities between the reference images stored in thereference image storage section and representative three-dimensionalobject models stored in the representative three-dimensional objectmodel storage section, in a reference image matching result storagesection; a result matching step of extracting the reference imagesimilar to the input image on the basis of similarities between theinput image and the representative three-dimensional object modelscalculated at the image matching step and similarities between thereference images and the representative three-dimensional object modelsstored in the reference image matching result storage section; a step ofstoring reference three-dimensional object models associated with thereference images stored in the reference image storage section, in areference three-dimensional object model storage section; a second imagegeneration step of obtaining reference three-dimensional object modelsassociated with reference images extracted at the result matching step,from the reference three-dimensional object model storage section, andgenerating at least one second comparison image close in input conditionto the input image every reference three-dimensional object model on thebasis of the obtained reference three-dimensional object models; and asecond image matching step of calculating similarities between the inputimage and second comparison images generated at the second imagegeneration step, selecting a maximum similarity from among secondcomparison images associated with each of the referencethree-dimensional object models, and regarding the maximum similarity asa similarity between the input image and the reference three-dimensionalobject model.
 48. The image matching program according to claim 47, theimage matching method further comprising: a three-dimensional objectmodel registration step of registering representative three-dimensionalobject models in the representative three-dimensional object modelstorage section; a reference image registration step of registeringreference images in the reference image storage section; and a referenceimage matching result update step of conducting calculation of thesimilarities using the image matching means, on a combination of areference image and a representative three-dimensional object modelnewly generated by registration, when a new representativethree-dimensional object model is registered in the representativethree-dimensional object model storage section at the three-dimensionalobject model registration step, or when a new reference image isregistered in the reference image storage section at the reference imageregistration step, and adding a result of the calculation to results inthe reference image matching result storage section; and athree-dimensional object model generation step of, in response toregistration of a similarity between the reference image and therepresentative three-dimensional object model in the reference imagematching result storage section conducted at the reference imagematching result update step, generating the reference three-dimensionalobject model associated with the reference image by combining therepresentative three-dimensional object models stored in therepresentative three-dimensional object model storage section on thebasis of the similarity, and registering the generated referencethree-dimensional object model in the reference three-dimensional objectmodel storage section.
 49. The image matching program according to claim48, wherein at the three-dimensional object model generation step, areference three-dimensional object model associated with each referenceimage is generated by combining representative three-dimensional objectmodels stored in the representative three-dimensional object modelstorage section every partial region, on the basis of similaritiesobtained every partial region between reference images stored in thereference image storage section and representative three-dimensionalobject models stored in the representative three-dimensional objectmodel storage section, and the generated reference three-dimensionalobject model is registered in the reference three-dimensional objectmodel storage section.
 50. The image matching program according to claim47, wherein at the image matching step, a similarity between the inputimage and a representative three-dimensional object model is calculatedevery partial region, the reference image matching result storagesection stores similarities between the reference images stored in thereference image storage section and representative three-dimensionalobject models stored in the representative three-dimensional objectmodel storage section, every partial region, and at the result matchingstep, the reference image similar to the input image is extracted on thebasis of similarities between the input image and the representativethree-dimensional object models calculated by the image matching meansevery partial region and similarities between the reference images andthe representative three-dimensional object models, stored in thereference image matching result storage section every partial region.51. The image matching program according to claim 47, wherein at theresult matching step, similarities between similarities between theinput image and the representative three-dimensional object models andsimilarities between the reference images and the representativethree-dimensional object models are calculated, and in the calculation,the resultant similarities are provided with weights on the basis ofcandidate precedence of similarities between the input image and thecomparison images and the representative three-dimensional objectmodels.
 52. The image matching program according to claim 41, the imagematching method comprising: an image input step of inputting the inputimage; a step of storing a plurality of representative three-dimensionalobject models in a representative three-dimensional object model storagesection an image generation step of generating at least one comparisonimage close in input condition to the input image every representativethree-dimensional object model on the basis of the representativethree-dimensional object models stored in the representativethree-dimensional object model storage section; an image matching stepof calculating a similarity between the input image and each of thecomparison images generated at the image generation step, selecting amaximum similarity with respect to comparison images associated witheach representative three-dimensional object model, and regarding themaximum similarity as a similarity between the input image and therepresentative three-dimensional object model; a step of storing thereference images of objects in a reference image storage section; a stepof storing similarities between the reference images stored in thereference image storage section and representative three-dimensionalobject models stored in the representative three-dimensional objectmodel storage section, in a reference image matching result storagesection; a result matching step of extracting the reference imagesimilar to the input image on the basis of similarities between theinput image and the representative three-dimensional object modelscalculated at the image matching step and similarities between thereference images and the representative three-dimensional object modelsstored in the reference image matching result storage section; a step ofstoring reference three-dimensional object models associated with thereference images stored in the reference image storage section, in areference three-dimensional object model storage section; an imageconversion step of obtaining reference three-dimensional object modelsassociated with reference images extracted at the result matching step,from the reference three-dimensional object model storage section,squaring an input condition of the input image with that of thereference image extracted at the result matching step by converting theinput image and/or the reference image extracted at the result matchingstep, on the basis of the obtained reference three-dimensional objectmodels, and generating partial images respectively of the input imageand the reference image squared in input condition with each other; anda partial image matching step of calculating a similarity between thepartial image of the input image and the partial image of the referenceimage generated at the image conversion step.
 53. The image matchingprogram according to claim 52, the image matching method furthercomprising: a three-dimensional object model registration step ofregistering representative three-dimensional object models in therepresentative three-dimensional object model storage section; areference image registration step of registering reference images in thereference image storage section; and a reference image matching resultupdate step of conducting calculation of the similarities at the imagematching step, on a combination of a reference image and arepresentative three-dimensional object model newly generated byregistration, when a new representative three-dimensional object modelis registered in the representative three-dimensional object modelstorage section at the three-dimensional object model registration step,or when a reference image is registered in the reference image storagesection at the reference image registration step, and adding a result ofthe calculation to results in the reference image matching resultstorage section; and a three-dimensional object model generation stepof, in response to registration of a similarity between the referenceimage and the representative three-dimensional object model in thereference image matching result storage section conducted at thereference image matching result update step, for generating thereference three-dimensional object model associated with the referenceimage by combining the representative three-dimensional object modelsstored in the representative three-dimensional object model storagesection on the basis of the similarity, and registering the generatedreference three-dimensional object model in the referencethree-dimensional object model storage section.
 54. The image matchingprogram according to claim 53, wherein at the three-dimensional objectmodel generation step, a reference three-dimensional object modelassociated with each reference image is generated by combiningrepresentative three-dimensional object models stored in therepresentative three-dimensional object model storage section everypartial region, on the basis of similarities obtained every partialregion between reference images stored in the reference image storagesection and representative three-dimensional object models stored in therepresentative three-dimensional object model storage section, and thegenerated reference three-dimensional object model is registered in thereference three-dimensional object model storage section.
 55. The imagematching program according to claim 52, wherein at the image matchingstep, a similarity between the input image and a representativethree-dimensional object model is calculated every partial region, thereference image matching result storage section stores similaritiesbetween the reference images stored in the reference image storagesection and representative three-dimensional object models stored in therepresentative three-dimensional object model storage section, everypartial region, and at the result matching step, the reference imagesimilar to the input image is extracted on the basis of similaritiesbetween the input image and the representative three-dimensional objectmodels calculated at the image matching step every partial region andsimilarities between the reference images and the representativethree-dimensional object models, stored in the reference image matchingresult storage section every partial region.
 56. The image matchingprogram according to claim 52, wherein at the result matching step,similarities between similarities between the input image and therepresentative three-dimensional object models and similarities betweenthe reference images and the representative three-dimensional objectmodels are calculated, and in the calculation, the resultantsimilarities are provided with weights on the basis of candidateprecedence of similarities between the input image and the comparisonimages and the representative three-dimensional object models.
 57. Theimage matching program according to claim 39, wherein the object is ahuman face.